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This paper describes the use of both quantitative and 
qualitative methods to evaluate a project designed to define a state system 
of care for meeting the needs of people with concurrent disorders of mental 
illness and substance abuse. The evaluation was intended to develop a 
feedback system to provide program implementers with information about the 
mechanics and processes of the project’s panel and to monitor the changing 
level of trust and cohesiveness among panel participants. A further goal of 
the evaluation was to illustrate and elucidate the key issues impeding and 
facilitating the adoption of best practices in treating the population of 
interest. Data collection strategies included participant observation, 
telephone interviews, document analysis, and meeting evaluation forms 
completed by participants at each session. From these sources, 17 themes 
related to the panel process emerged. Data from the qualitative interviews 
produced a richer and more contextual perspective from which to evaluate the 
degree and quality of consensus among the panel participants. Combining 
quantitative and qualitative approaches allowed for triangulation and 
complementarity in the evaluation process. (SLD) 
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The utilization of mixed methods to study consensus building in dual 

diagnosis treatment 

Michael S. Shafer & Denver Lewellen 
University of Arizona 
School of Public Administration & Policy 

Background & Context 

There is a growing awareness of the rates of co-occurrence of substance abuse among 
persons with severe mental disorders such as schizophrenia. Commonly accepted 
prevalence rates estimates the lifetime rates of substance abuse among persons with 
serious mental illness to be between 40% and 60% (Drake, Mercer-McFadden, Mueser, 
McHugo, & Bond, 1998). A number of issues have been identified that impede access to 
effective services for individuals experiencing co-disorders (Ridgley, Goldman, & 
Willenbring, 1990). These include separate funding streams for mental health and 
substance abuse treatment services, different theoretical-practice orientations (abstinence 
versus harm reduction), and different staffing models (peer staff traditions in substance 
abuse treatment versus credentialed staff in mental health treatment). Additionally, the 
growing transformation of community mental health services into managed care creates 
implications and barriers to effective integrated services. 

Due to funding from the Substance Abuse and Mental Health Services Administration 
(SAMHSA), the Arizona Department of Health Services initiated a one-year consensus 
building project to define a state system of care for meeting the needs of persons with co- 
occurring disorders of mental illness and substance abuse. A thirty-five member cross- 
sectional panel was drawn from Phoenix and Tucson, the two largest urban populations in 
Arizona. This panel included representatives from state agencies, local regional 
behavioral health authorities, residential treatment providers, advocacy organizations, and 
case management agencies. Individuals included consumers, case managers, program 
managers, psychiatrists, advocates, and family members. The panel met on a monthly 
basis, and supported by a meeting facilitator, engaged in a 9-step strategic planning 
process. Additionally, the panel participated in periodic “knowledge exchange” sessions 
at which nationally prominent experts in the field of dual diagnosis treatment presented to 
the panel participants and typically followed with less formal exchanges, dialogues and 
discussions. 

Evaluation Design 

A combination of quantitative and qualitative methods were employed to evaluate this 
project. Our evaluation design was developed to address three key interests of the project 
team. First, we wanted to develop a feedback system that would provide the program 
implementers, most notably, the meeting facilitator, with information regarding the 
mechanics and processes of the group, including meeting effectiveness, representation 
and participation, and decision-making within the group. Second, we wanted to monitor 
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the changing level of cohesion and trust among the panel participants and to study these 
changes relative to the activities and issues being addressed by the panel. Finally, we 
wanted to illustrate and better understand the key issues impeding and facilitating the 
adoption of best practices in treating the population of interest. 

Data Collection Strategies 

• Participant Observation . Either the first or second author was in attendance at 
each of the panel meetings. The evaluators’ presence at the meetings was 
explained at the first series of meetings and participants came to accept our 
presence and lack of involvement in the proceedings of the meetings. Informal 
networking and conversation would ensue between the evaluators and the panel 
participants during breaks and at lunch. These informal discussions and ongoing 
observation of the group afforded insight to understanding the dynamics of the 
group and the nature and quality of the participation and affiliation among and 
between the panel participants. 

• Telephone Interviews . Interviews with panel participants were conducted by 
phone (two were conducted in person) over a period of 8 months. These 
interviews were conducted the second author, who also attended a majority of the 
panel meetings. The data from these interviews were confidential, accessible 
only to the two evaluators and the group facilitator. The interviews generally took 
between half an hour and an hour, sometimes more or less depending upon the 
stakeholders’ available time. No formal questionnaire was developed for the 
individual stakeholder interviews. Rather, a checklist of general questions was 
gone over during each interview and stakeholders were encouraged to interpret 
this checklist in ways they thought were most appropriate. Subjects on the 
checklist included: 

• Stakeholder qualifications and histories of experience with dually 
diagnosed populations 

• Current services in Arizona 

• Consensus 

• Panel's progress 

• Panel representation 

• Implementation concerns 

• Recommendations for future monthly meetings 

• Document Analysis . All key documents developed by the Consensus Panel were 
reviewed by the first and second author to identify recurrent themes, primary foci, 
panel principles and values, and outcomes of the group. These documents included 
the following: 

y Groundrules: adopted by the panel early in the meeting schedule and 

delineated the manner in which the panel meetings would operate and 

consensus would be defined. 
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^ Principles: adopted by the panel approximately mid three-quarters through 
the meeting schedule that articulates principles that the Panel endorsed as 
guiding best practice in treating persons with dial disorders. 

S Goals and Objectives: adopted by the panel and defining the outcomes, 

products, timelines and responsibilities that the panel would follow in carrying 
out their activities. 

Outcomes: adopted by the panel and delineating anticipated/desired outcomes 
for programs/sy stems and individuals that result from integrated treatment 
which the Panel endorsed as the goals of integrated services implementation 
in Arizona. 

S Sub-Committee Reports: in implementing the workplan that they developed, 
the panel divided into three sub-committees or work groups. These included: 
funding, policies/procedures, and staff development/training. All reports, 
proposals, and correspondence of the sub-committee were reviewed for 
recurrent themes and cross-committee issues. 

• Meeting Evaluation Forms . A 23-item questionnaire was distributed and 

collected at the conclusion of each meeting. These items used a 5-point Likert 
scale (anchored at Strongly Disagree/Strongly Agree) to solicit participants’ 
responses to a series of statements assessing four constructs: 

• Group Cohesion 

• Participation 

• Meeting Effectiveness 

• Group Consensus 

Additionally, the questionnaire contained open-ended questions soliciting 
feedback about what the participants liked best about the meeting, liked least 
about the meeting, and recommendations for future meetings. Simple univariate 
summary reports of each meetings’ completed evaluation forms were forwarded 
to the meeting facilitator as an ongoing formative feedback loop. 

Results & Findings 

Near completion of the one year consensus building period, qualitative interviews have 
been conducted with nearly all of the panel participants. Following these interviews, the 
write-ups for each interview were compiled in chronological date order into one 
document. Manual coding from this document ensued, using topic cards with subjects 
derived from frequency of responses. From the small set of subjects on the interview 
checklist, a total of 1 7 specific themes emerged. The following histogram displays the 
relative frequency of these topics. 
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Following is a review of two of these themes; Consensus and Panel's Progress. These 
two themes were selected for illustrative purposes only for demonstrating the integration 
of qualitative and quantitative data. These themes have been selected for review because 
they perhaps best indicate the range of ideas, perceptions, hopes and concerns about the 
panel process, it's subject matter and it's possible outcomes. 

Consensus 

Four items from our meeting evaluation questionnaire attempted to assess participants’ 
perceptions regarding the degree of consensus that had been achieved by the group. 
These four items read as follows and were rated individually on a five-point Likert scale: 

S The group used effective decision making techniques 
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S At today's meeting, there was agreement among the group about what are the 
major barriers 

S At today's meeting, there was agreement among the group about what an 
improved method of serving this population should be 
S At today's meeting, there was agreement among the group what action needs to be 
taken to implement an improved system of serving this population. 




As these data reveal, responses to all four statements were in general agreement, 
suggesting good consensus and effective decision making. All four items showed 
improvements over time, as the group continued to meet and as it began to move from 
more conceptual activities to more concrete actions. 

Data collected from our qualitative interviews, however, provided a much richer and 
contextual perspective from which to evaluate the degree and quality of consensus among 
the panel participants. At the beginning of the panel, the stakeholders decided that the 
idea of "consensus" as it related to the integrated treatment panel was not going to refer to 
a strident concept of unanimous, absolute agreement. As the group's facilitator puts it, "it 
meant that either you agree with something or you can live with it." In spite of this early 
agreement, however, panel members did express some concern about the definition of 
consensus, and about whether or not consensus was truly being reached. For example, 
the sole panel member representing the "front line" of case management, expressed her 
concern that issues were being resolved too quickly, or that they were really not being 
resolved at all. According to this participant, it was this faulty way of deciding things 
which resulted in the determination that a particular Hispanic professional was brought 
onto the panel. She said that there were other, equally interesting candidates for the slot, 
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but all it took was the agreement of a few individuals on the panel and the case was 
closed. This issue of voting was also of concern to a program manager: 



It (the panel) doesn't vote on anything. The only thing we voted on was who was 
going to be on the panel and we embarrassed ourselves doing that. When one 
person disagrees, we drop it and go on to something else. We at least need to 
have clarity and have a straw vote before we let it go. The way it is now is that if 
we're sick of it we move on. 



Several other panel members expressed concerns over the fact that, regularly, certain 
panel members remained quiet throughout the meetings, expressing neither agreement or 
disagreement. A psychiatrist summed up the concern: 



Should Silence be taken as consent? Or is it something else? It really doesn't 
matter. The question is - when should we have a vote? 



In one outstanding instance, there emerged a difference of opinion on two counts. The 
first was in regards to the full definition of "SMI", related to what range of persons with 
mental health conditions would ultimately be able to receive services; the second was in 
regards to whether or not consensus had been reached on what that definition was. One 
particular panel member never felt satisfied with the definition and had the impression 
that the panel would return to the issue and work it out. By all other accounts, a 
consensus had been reached on the definition and the other panel members had no 
intention of ever going back to it. Repeatedly, this miscommunication reared its head in 
the monthly meetings, sometimes creating a sense of exasperation and hostility. 

Panel's Progress 

No items from the Meeting Questionnaire assessed the issue of the progress of the panel, 
although one statement asked participants to respond to the statement, “The group 
accomplished what is set out to do”. Average scores for this item ranged from 3.87 
(SD=.69) at the initial meeting, to 4.47 (SD=.64) at the most recent meeting. However, 
telephone interviews provided a more graphic and realistic perspective on the panelist’s 
views and concerns about the panel’s progress. Ideas about the progress of the panel, put 
in terms of the ability of the panel to meet its deadlines, varied more than any other 
theme. Initially, stakeholders were optimistic, with some expressing feelings of pleasant 
surprise, over the forward movements of the panel during the first several meetings. 

Many stakeholders were also panel members of a similar panel and these other 
affiliations provided a basis from which to compare. Early criticisms of the panel by 
stakeholders generally related to what were seen as unnecessary time allotted to getting 
grammar and wording of sentences just right. Certain stakeholders suspected that this 
time was spent in lieu of more important work and this came through in the interviews in 
spite of generally favorable reviews of the panel's progress: 
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There is too much attention paid to that wordsmithing crap. Paring down is not 
our job. The funders will do that anyways. But it is easier for people to do this 
than to deal with what really needs to be done. 



By July, concern about the panel's progress shifted. Panel members began expressing 
serious concern about the fact that time was marching on and the core principles had not 
been completed. 



We have a lot to do and it's very hard to work in these subgroups between the 
meetings. We've gotten bogged down and I'm not 100% sure where we are. If 
Linda or someone could map things out for us at the next meeting that would be 
very helpful. But people are all pretty strapped for time. It would be nice for the 
group if it could keep meeting - after the panel ends — to help keep things on 
track. 



The concern over the timeline was also shared by the group's facilitator: 



We needed to have started the subcommittee work earlier, what they are doing 
now could have been done a couple of months ago. Having said this, I believe 
that the time spent so far has been very valuable. But the group has only 4 
meetings left and that's not a lot of time and I'm not comfortable with everything 
we need to do in that time. I wouldn't want to take away for the education aspect 
of the panel, but we have a lot to do. This is different from the TOPPS panel 
because on that panel the tasks are very specific. This is much more 
developmental. I hope that people will know that this kind of change doesn 't 
happen overnight. The plan is for the system to come together over time. 



Key Implications 

In summary, we utilized a mixed methods approach to conduct a formative evaluation of 
an initiative to develop consensus regarding treatment services to persons with mental 
illnesses and co — occurring substance abuse disorders. In this presentation, very 
preliminary findings and a limited amount of the data that have been collected and 
analyzed were presented to highlight the integration of qualitative and quantitative 
methodologies. By combining both approaches, our evaluation of this process was 
enriched and allowed for at two of the five purposes that Greene, Caracelli, & Graham 
(1989) lay out for mixed method evaluation designs. The examples provided above 
illustrate both the purpose of triangulation and complementarity. Three key implications 
of this evaluation warrant discussion: 

■S Linking evaluative feedback to group facilitation processes: in this project, 
we established feedback loops to the meeting facilitator about the information 
we collecting through our interviews and meeting questionnaires. This proved 
to be an invaluable source of rich information that allowed the facilitator to 
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adjust her style and group process to respond to the concerns, criticisms, and 
recommendations from the panel members that were not articulated within the 
larger group setting. 

S Issues and challenges of representation: one of the greatest challenges of this 
type of collaborative process, is ensuring adequate and appropriate 
representation. In this initiative, persons of color and persons with mental 
illness and substance abuse disorders were not well represented, and for the 
most part, those that were invited to participate, attended a meeting or two and 
then dropped out. As such, greater attention needs to be paid to ensure the 
participation of a representative constituent. 

S Continuing Efforts of the Panel: the panel continues to meet on a monthly 
basis and has organized itself into three sub-committee: funding; policies and 
procedures; and staff development and competencies. In December of this 
year, the panel will make a formal presentation to leading state agency heads. 
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